agent/proxy: cancel the previous auto-auth token when re-authenticating - #32060

Fixes #25712.

The problem

When auto-auth re-authenticates, RegisterAutoAuthToken adds the new token to the lease cache but never removes the one it replaces:

https://github.com/hashicorp/vault/blob/main/command/agentproxyshared/cache/lease_cache.go#L1763-L1819

If the token is already indexed the call is a no-op; otherwise a new index is created with a context derived from the cache's base context. The previous token's index stays in the cache and its context is never cancelled.

Vault revokes a token's child leases when that token expires, so every lifetime watcher derived from the old token can only ever get permission denied back from sys/leases/renew. Those watchers keep retrying until each one individually gives up.

This is what #25712 reports, and it is still reproducible on v2.0.3.

Reproduction and measurements

Reproduced deliberately on Vault Agent 1.14.0 by setting a Kubernetes auth role's token_max_ttl to 300s and a database secrets engine lease TTL of 600s, so the lease outlives its parent token. Three agent pods, one hour, 16 auto-auth generations each.

From the agent's own debug logs, for one rotation:

15:29:28   storm begins server-side (token already dead)
15:29:38   lifetime watcher done channel triggered
15:29:38   authentication successful, sending token to sinks   <- new token, 10s later
15:29:38   initiating renewal db-creds                         <- new watchers started
   ...     storm continues against the OLD token
15:30:54   renewal halted; evicting from cache  db-creds          (+76s)
15:31:20   renewal halted; evicting from cache  sys/leases/renew  (+102s)

The agent gets a healthy replacement token within 10 seconds. The orphaned watchers keep going for another 76 and 102 seconds.

From the Vault audit log over the same window, one token/lease pair, one pod:

34 088 failing sys/leases/renew in ~150s   (~227 req/s average, ~356 req/s peak)

Across three pods rotating out of phase this sustained roughly 1 100 req/s of requests that could not succeed, in ~2 minute bursts on every rotation, against a single-leader cluster. The commenter on #25712 reports the same magnitude — "from 30 request per seconds to over 1100 for 1 hour and 10 minutes" — on GCP rather than AWS, so this does not look auth-method specific.

Worth noting for anyone searching: none of this appears in the agent logs at info. The lease cache logs renewal activity at debug, and the failing requests themselves are never logged at all — only the eventual renewal halted; evicting from cache is.

The change

RegisterAutoAuthToken now records the current auto-auth token and cancels the context of the one it replaces.

A lease's renewal context is already derived from its token's context:

https://github.com/hashicorp/vault/blob/main/command/agentproxyshared/cache/lease_cache.go#L544

so cancelling the token's context cascades to every lease obtained with it — the watchers stop and the entries are evicted through the existing defer in startRenewing. The cascade mechanism was already there; nothing was calling it on rotation.

Three details worth pointing at during review:

• The cancellation runs before the "already cached" short-circuit. That short-circuit is precisely the path taken after a restart with a persistent cache: restoreTokens has already re-added the previous token, so auto-auth's first registration finds it and returns early. Recording the token only after that check would leave the tracker empty exactly in the scenario where the stale watchers are most likely to be running.
• Empty tokens are ignored. The sink can be handed an empty token while auto-auth is shutting down, and that must not cancel the live token's context. This has a second effect worth flagging since it is not what the PR title is about: previously an empty write built a junk TokenType index and BoltStorage.Set then overwrote the AutoAuthToken meta key with it, so on the next start previousToken came back empty and the agent did a full re-authentication instead of lookup-self with the persisted token. Skipping the empty registration keeps that meta key intact.
• Named cancelPreviousAutoAuthToken, not evict... — the index itself stays in the cache with a cancelled context, which is what the "token" branch of handleCacheClear already does. Only the derived leases are evicted, by startRenewing's defer.

Guarded by a dedicated mutex rather than the existing c.l, since c.l protects baseCtxInfo and createCtxInfo takes an RLock on it.

Trade-off to weigh

This cancels on any auto-auth token change, not only on expiry. auth.go also re-authenticates when the auth method reports new credentials (credCh) or after a transient lifetime-watcher error, and in those cases the previous token may still be valid. Its leases are then dropped from the cache and re-fetched on next use instead of being kept renewed. Nothing is revoked and nothing leaks — the failure direction is "less cached material" — but it is a behaviour change and it is documented in the function comment.

The narrower alternative would be to drive the cancellation from a signal that the token is genuinely gone (the InvalidToken / permanent-lookup-failure paths) rather than from "a different token arrived". Happy to rework it that way if you would prefer; it did not seem worth the extra coupling for a case where the old token is almost always dead.

Only the auto-auth token is affected. Client tokens that applications send through the proxy are indexed separately and are untouched.

This does not change the api.LifetimeWatcher backoff fixed in #26383; it removes the need to rely on it for this particular case.

Testing

Three tests, measured against main:

The first two are regression tests for the reported bug — the second specifically covers the persistent-cache restore path. The third covers the empty-token guard and the requirement that an empty write leave the tracked token in place.

On main:

--- FAIL: TestLeaseCache_RegisterAutoAuthToken_CancelsPreviousToken (2.00s)
    lease_cache_test.go:1713: context of the previous auto-auth token was not cancelled
--- FAIL: TestLeaseCache_RegisterAutoAuthToken_CancelsAfterRestore (2.00s)
    lease_cache_test.go:1763: context of the restored auto-auth token was not cancelled on re-authentication

go test -race ./command/agentproxyshared/cache/... passes: 146 tests across 4 packages. go vet reports only the two pre-existing findings in listener.go and static_secret_cache_updater.go, both untouched here.